Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training

TL;DR

This paper introduces KoBo, a novel framework that enhances medical vision-language pre-training by integrating clinical knowledge, improving performance across various tasks without extensive annotations.

Contribution

The paper proposes a knowledge-boosting framework that incorporates clinical knowledge into contrastive pre-training, addressing semantic overlap and shifting issues in medical AI.

Findings

Improved performance on classification, segmentation, retrieval, and semantic relatedness tasks.

Achieves comparable or better results in zero-shot and few-shot settings.

Validated effectiveness across eight diverse medical tasks.

Abstract

The foundation models based on pre-training technology have significantly advanced artificial intelligence from theoretical to practical applications. These models have facilitated the feasibility of computer-aided diagnosis for widespread use. Medical contrastive vision-language pre-training, which does not require human annotations, is an effective approach for guiding representation learning using description information in diagnostic reports. However, the effectiveness of pre-training is limited by the large-scale semantic overlap and shifting problems in medical field. To address these issues, we propose the Knowledge-Boosting Contrastive Vision-Language Pre-training framework (KoBo), which integrates clinical knowledge into the learning of vision-language semantic consistency. The framework uses an unbiased, open-set sample-wise knowledge representation to measure negative sample noise and supplement the correspondence between vision-language mutual information and clinical knowledge. Extensive experiments validate the effect of our framework on eight tasks including classification, segmentation, retrieval, and semantic relatedness, achieving comparable or better performance with the zero-shot or few-shot settings. Our code is open on https://github.com/ChenXiaoFei-CS/KoBo.