Abnormality-Driven Representation Learning for Radiology Imaging

TL;DR

This paper introduces CLEAR, a novel framework for radiology imaging that leverages abnormality-driven contrastive learning to create efficient, task-agnostic representations, outperforming existing models in clinical tasks.

Contribution

The paper proposes a new framework and a lesion-enhanced contrastive learning method for radiology images, addressing the lack of task-agnostic models in 3D radiology imaging.

Findings

CLEAR outperforms state-of-the-art models in clinical tasks.

LeCL improves representation quality for radiology images.

The approach is more compute- and data-efficient.

Abstract

To date, the most common approach for radiology deep learning pipelines is the use of end-to-end 3D networks based on models pre-trained on other tasks, followed by fine-tuning on the task at hand. In contrast, adjacent medical fields such as pathology, which focus on 2D images, have effectively adopted task-agnostic foundational models based on self-supervised learning (SSL), combined with weakly-supervised deep learning (DL). However, the field of radiology still lacks task-agnostic representation models due to the computational and data demands of 3D imaging and the anatomical complexity inherent to radiology scans. To address this gap, we propose CLEAR, a framework for radiology images that uses extracted embeddings from 2D slices along with attention-based aggregation for efficiently predicting clinical endpoints. As part of this framework, we introduce lesion-enhanced contrastive learning (LeCL), a novel approach to obtain visual representations driven by abnormalities in 2D axial slices across different locations of the CT scans. Specifically, we trained single-domain contrastive learning approaches using three different architectures: Vision Transformers, Vision State Space Models and Gated Convolutional Neural Networks. We evaluate our approach across three clinical tasks: tumor lesion location, lung disease detection, and patient staging, benchmarking against four state-of-the-art foundation models, including BiomedCLIP. Our findings demonstrate that CLEAR using representations learned through LeCL, outperforms existing foundation models, while being substantially more compute- and data-efficient.