Multi-Modal Foundation Models for Computational Pathology: A Survey

TL;DR

This survey reviews recent advances in multi-modal foundation models for computational pathology, focusing on integrating visual data with text, knowledge, and molecular profiles to improve analysis of histopathological images.

Contribution

It categorizes and analyzes 32 state-of-the-art multi-modal models, datasets, and tasks, providing a comprehensive taxonomy and highlighting future research directions.

Findings

Classification of models into vision-language, vision-knowledge graph, and vision-gene expression paradigms.

Analysis of 28 multi-modal datasets for pathology applications.

Identification of key challenges and future research directions.

Abstract

Foundation models have emerged as a powerful paradigm in computational pathology (CPath), enabling scalable and generalizable analysis of histopathological images. While early developments centered on uni-modal models trained solely on visual data, recent advances have highlighted the promise of multi-modal foundation models that integrate heterogeneous data sources such as textual reports, structured domain knowledge, and molecular profiles. In this survey, we provide a comprehensive and up-to-date review of multi-modal foundation models in CPath, with a particular focus on models built upon hematoxylin and eosin (H&E) stained whole slide images (WSIs) and tile-level representations. We categorize 32 state-of-the-art multi-modal foundation models into three major paradigms: vision-language, vision-knowledge graph, and vision-gene expression. We further divide vision-language models into non-LLM-based and LLM-based approaches. Additionally, we analyze 28 available multi-modal datasets tailored for pathology, grouped into image-text pairs, instruction datasets, and image-other modality pairs. Our survey also presents a taxonomy of downstream tasks, highlights training and evaluation strategies, and identifies key challenges and future directions. We aim for this survey to serve as a valuable resource for researchers and practitioners working at the intersection of pathology and AI.