Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration

TL;DR

This paper introduces PLACE, a novel framework for medical visual representation learning that emphasizes pathological-level cross-modal alignment and correlation exploration, improving performance across multiple medical imaging tasks without extra annotations.

Contribution

The paper proposes a new pathological-level cross-modal alignment method and a correlation exploration proxy task, enhancing medical visual representations without relying on external disease annotations.

Findings

Achieves state-of-the-art results on classification tasks

Improves image-to-text retrieval performance

Enhances semantic segmentation and object detection accuracy

Abstract

Learning medical visual representations from image-report pairs through joint learning has garnered increasing research attention due to its potential to alleviate the data scarcity problem in the medical domain. The primary challenges stem from the lengthy reports that feature complex discourse relations and semantic pathologies. Previous works have predominantly focused on instance-wise or token-wise cross-modal alignment, often neglecting the importance of pathological-level consistency. This paper presents a novel framework PLACE that promotes the Pathological-Level Alignment and enriches the fine-grained details via Correlation Exploration without additional human annotations. Specifically, we propose a novel pathological-level cross-modal alignment (PCMA) approach to maximize the consistency of pathology observations from both images and reports. To facilitate this, a Visual Pathology Observation Extractor is introduced to extract visual pathological observation representations from localized tokens. The PCMA module operates independently of any external disease annotations, enhancing the generalizability and robustness of our methods. Furthermore, we design a proxy task that enforces the model to identify correlations among image patches, thereby enriching the fine-grained details crucial for various downstream tasks. Experimental results demonstrate that our proposed framework achieves new state-of-the-art performance on multiple downstream tasks, including classification, image-to-text retrieval, semantic segmentation, object detection and report generation. Code is available at https://github.com/Markin-Wang/PLACE.