Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework

TL;DR

This paper introduces a multi-aspect vision-language pre-training framework that decomposes disease descriptions into fundamental aspects, improving pathology detection accuracy for both known and unknown diseases by aligning images with detailed textual features.

Contribution

The novel framework dissects disease descriptions into aspects using prior knowledge and a large language model, enhancing image-text alignment and detection performance in medical VLP.

Findings

Improves accuracy by up to 8.56% on seen diseases.

Enhances unseen disease detection by 17.26%.

Outperforms recent methods across seven datasets.

Abstract

Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL.