A multi-modal vision-language model for generalizable annotation-free pathology localization

TL;DR

This paper introduces AFLoc, a multi-modal vision-language model that achieves generalizable, annotation-free pathology localization across various medical imaging modalities, outperforming existing methods and even surpassing human benchmarks.

Contribution

The paper presents a novel contrastive learning framework that aligns multi-granularity medical concepts with image features, enabling pathology localization without expert annotations.

Findings

Outperforms state-of-the-art in localization and classification

Demonstrates strong generalization across modalities

Surpasses human benchmarks in pathology localization

Abstract

Existing deep learning models for defining pathology from clinical imaging data rely on expert annotations and lack generalization capabilities in open clinical environments. Here, we present a generalizable vision-language model for Annotation-Free pathology Localization (AFLoc). The core strength of AFLoc is extensive multi-level semantic structure-based contrastive learning, which comprehensively aligns multi-granularity medical concepts with abundant image features to adapt to the diverse expressions of pathologies without the reliance on expert image annotations. We conduct primary experiments on a dataset of 220K pairs of image-report chest X-ray images and perform validation across eight external datasets encompassing 34 types of chest pathologies. The results demonstrate that AFLoc outperforms state-of-the-art methods in both annotation-free localization and classification tasks. Additionally, we assess the generalizability of AFLoc on other modalities, including histopathology and retinal fundus images. We show that AFLoc exhibits robust generalization capabilities, even surpassing human benchmarks in localizing five different types of pathological images. These results highlight the potential of AFLoc in reducing annotation requirements and its applicability in complex clinical environments.