Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis

TL;DR

This paper introduces ConcepPath, a novel MIL framework that aligns histopathology images with expert-derived knowledge concepts using GPT-4 and vision-language models, significantly improving cancer subtyping accuracy.

Contribution

ConcepPath uniquely integrates human expert concepts with learnable concepts in MIL, leveraging GPT-4 and vision-language models for precise histopathology analysis.

Findings

Outperforms previous state-of-the-art methods in cancer subtyping tasks.

Effectively incorporates expert knowledge into image analysis.

Enhances interpretability of histopathology models.

Abstract

Due to the large size and lack of fine-grained annotation, Whole Slide Images (WSIs) analysis is commonly approached as a Multiple Instance Learning (MIL) problem. However, previous studies only learn from training data, posing a stark contrast to how human clinicians teach each other and reason about histopathologic entities and factors. Here we present a novel knowledge concept-based MIL framework, named ConcepPath to fill this gap. Specifically, ConcepPath utilizes GPT-4 to induce reliable diseasespecific human expert concepts from medical literature, and incorporate them with a group of purely learnable concepts to extract complementary knowledge from training data. In ConcepPath, WSIs are aligned to these linguistic knowledge concepts by utilizing pathology vision-language model as the basic building component. In the application of lung cancer subtyping, breast cancer HER2 scoring, and gastric cancer immunotherapy-sensitive subtyping task, ConcepPath significantly outperformed previous SOTA methods which lack the guidance of human expert knowledge.