Anatomical Structure-Guided Medical Vision-Language Pre-training

TL;DR

This paper introduces an Anatomical Structure-Guided framework for medical vision-language pre-training that enhances interpretability and clinical relevance by leveraging anatomical parsing and fine-grained alignment.

Contribution

It proposes a novel anatomical structure-guided approach with report parsing, anatomical region-sentence alignment, and image-tag recognition to improve medical visual representation learning.

Findings

Outperforms state-of-the-art methods on five benchmarks.

Enhances local interpretability and semantic alignment.

Improves downstream task performance.

Abstract

Learning medical visual representations through vision-language pre-training has reached remarkable progress. Despite the promising performance, it still faces challenges, i.e., local alignment lacks interpretability and clinical relevance, and the insufficient internal and external representation learning of image-report pairs. To address these issues, we propose an Anatomical Structure-Guided (ASG) framework. Specifically, we parse raw reports into triplets <anatomical region, finding, existence>, and fully utilize each element as supervision to enhance representation learning. For anatomical region, we design an automatic anatomical region-sentence alignment paradigm in collaboration with radiologists, considering them as the minimum semantic units to explore fine-grained local alignment. For finding and existence, we regard them as image tags, applying an image-tag recognition decoder to associate image features with their respective tags within each sample and constructing soft labels for contrastive learning to improve the semantic association of different image-report pairs. We evaluate the proposed ASG framework on two downstream tasks, including five public benchmarks. Experimental results demonstrate that our method outperforms the state-of-the-art methods.