Representation Learning with Semantic-aware Instance and Sparse Token Alignments

TL;DR

This paper introduces SISTA, a semantic-aware multi-level alignment framework for medical vision-language pre-training that improves representation quality by addressing false negatives and aligning image patches with relevant words.

Contribution

The paper proposes a novel multi-level alignment framework that enhances medical VLP by incorporating semantic correspondence and sparse token alignments, improving downstream task performance.

Findings

Improves transfer performance across multiple datasets and tasks.

Achieves significant gains in fine-grained tasks with limited labeled data.

Effectively reduces false negatives in contrastive learning.

Abstract

Medical contrastive vision-language pre-training (VLP) has demonstrated significant potential in improving performance on downstream tasks. Traditional approaches typically employ contrastive learning, treating paired image-report samples as positives and unpaired ones as negatives. However, in medical datasets, there can be substantial similarities between images or reports from different patients. Rigidly treating all unpaired samples as negatives, can disrupt the underlying semantic structure and negatively impact the quality of the learned representations. In this paper, we propose a multi-level alignment framework, Representation Learning with Semantic-aware Instance and Sparse Token Alignments (SISTA) by exploiting the semantic correspondence between medical image and radiology reports at two levels, i.e., image-report and patch-word levels. Specifically, we improve the conventional contrastive learning by incorporating inter-report similarity to eliminate the false negatives and introduce a method to effectively align image patches with relevant word tokens. Experimental results demonstrate the effectiveness of the proposed framework in improving transfer performance across different datasets on three downstream tasks: image classification, image segmentation, and object detection. Notably, our framework achieves significant improvements in fine-grained tasks even with limited labeled data. Codes and pre-trained models will be made available.