MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment

TL;DR

MAKE introduces a novel multi-aspect knowledge-enhanced vision-language pretraining framework that effectively integrates clinical knowledge and visual features for zero-shot dermatological diagnosis, surpassing existing models.

Contribution

The paper proposes a multi-aspect contrastive learning and fine-grained alignment approach to improve dermatological AI by incorporating structured clinical knowledge into vision-language pretraining.

Findings

Outperforms state-of-the-art models on eight dermatology datasets

Achieves significant improvements in zero-shot skin disease classification

Enhances concept annotation and cross-modal retrieval accuracy

Abstract

Dermatological diagnosis represents a complex multimodal challenge that requires integrating visual features with specialized clinical knowledge. While vision-language pretraining (VLP) has advanced medical AI, its effectiveness in dermatology is limited by text length constraints and the lack of structured texts. In this paper, we introduce MAKE, a Multi-Aspect Knowledge-Enhanced vision-language pretraining framework for zero-shot dermatological tasks. Recognizing that comprehensive dermatological descriptions require multiple knowledge aspects that exceed standard text constraints, our framework introduces: (1) a multi-aspect contrastive learning strategy that decomposes clinical narratives into knowledge-enhanced sub-texts through large language models, (2) a fine-grained alignment mechanism that connects subcaptions with diagnostically relevant image features, and (3) a diagnosis-guided weighting scheme that adaptively prioritizes different sub-captions based on clinical significance prior. Through pretraining on 403,563 dermatological image-text pairs collected from education resources, MAKE significantly outperforms state-of-the-art VLP models on eight datasets across zero-shot skin disease classification, concept annotation, and cross-modal retrieval tasks. Our code will be made publicly available at https: //github.com/SiyuanYan1/MAKE.