SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging

TL;DR

SparseContrast introduces a dynamic sparse attention mechanism into contrastive learning for medical imaging, significantly reducing computational costs while maintaining or improving diagnostic accuracy, especially in low-data scenarios.

Contribution

It presents a novel framework that adaptively trims attention maps using a saliency predictor, enhancing efficiency and accuracy across different model architectures in medical imaging.

Findings

Speeds up training and inference by up to 40% compared to dense attention methods.

Achieves comparable or superior disease detection performance with reduced computational resources.

Effective across both convolutional and transformer-based models.

Abstract

We propose SparseContrast, a new framework that merges dynamic sparse attention with contrastive learning for medical imaging, with a focus on chest X-ray disease detection in low-data settings. Traditional contrastive learning methods rely on dense attention mechanisms, which are computationally expensive and often process redundant regions in medical images. To resolve this, SparseContrast introduces a sparse attention mechanism that selectively concentrates on diagnostically pertinent areas, markedly decreasing computational burden without compromising accuracy. The framework adaptively trims attention maps in the training phase, directed by a compact saliency predictor which concurrently optimizes sparsity and feature quality. This method not only speeds up training and inference by as much as 40% relative to dense attention benchmarks but also boosts diagnostic accuracy by focusing on areas of clinical importance. Moreover, the approach remains indifferent to the selection of backbone architecture, which permits its application to both convolutional and transformer-based models. Experiments show SparseContrast attains comparable or better performance in disease identification tasks with greater efficiency relative to current approaches. The proposed framework delivers a practical approach for implementing contrastive learning in medical imaging settings with limited resources, where computational efficiency and diagnostic accuracy are paramount.