Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosis

TL;DR

This paper presents KEEP, a vision-language pathology model that integrates extensive medical knowledge graphs into pretraining, significantly improving cancer diagnosis accuracy especially for rare subtypes across multiple datasets.

Contribution

Introducing KEEP, a novel knowledge-enhanced pretraining framework that incorporates a large disease knowledge graph into vision-language models for pathology.

Findings

Outperforms existing models on 18 public benchmarks.

Achieves substantial improvements in rare cancer subtype diagnosis.

Effectively leverages disease ontology for semantic alignment.

Abstract

Vision-language foundation models have shown great promise in computational pathology but remain primarily data-driven, lacking explicit integration of medical knowledge. We introduce KEEP (KnowledgE-Enhanced Pathology), a foundation model that systematically incorporates disease knowledge into pretraining for cancer diagnosis. KEEP leverages a comprehensive disease knowledge graph encompassing 11,454 diseases and 139,143 attributes to reorganize millions of pathology image-text pairs into 143,000 semantically structured groups aligned with disease ontology hierarchies. This knowledge-enhanced pretraining aligns visual and textual representations within hierarchical semantic spaces, enabling deeper understanding of disease relationships and morphological patterns. Across 18 public benchmarks (over 14,000 whole-slide images) and 4 institutional rare cancer datasets (926 cases), KEEP consistently outperformed existing foundation models, showing substantial gains for rare subtypes. These results establish knowledge-enhanced vision-language modeling as a powerful paradigm for advancing computational pathology.