PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue

TL;DR

PRISM2 is a large multimodal foundation model for pathology that leverages clinical dialogue supervision to achieve high diagnostic accuracy and generalize across tasks, bridging human reasoning and AI capabilities.

Contribution

It introduces PRISM2, the largest histopathology dataset and a multimodal model trained on clinical dialogue, enhancing pathology AI's generalization and diagnostic performance.

Findings

PRISM2 matches or exceeds clinical-grade cancer detection performance.

PRISM2 achieves top results on various pathology tasks without additional training.

Task-specific fine-tuning with large datasets further improves performance.

Abstract

Recent rapid progress in the field of computational pathology has been enabled by foundation models. These models are beginning to move beyond encoding image patches towards whole-slide understanding but their clinical utility remains limited. In this work, we present PRISM2, a multimodal slide-level foundation model trained on data from 700,000 diagnostic specimen-report pairs, the largest vision (2.3 million whole slide images) and language (14M question-answer pairs) histopathology dataset to date. By learning through clinical-dialogue supervision, PRISM2 aligns histomorphologic features with the language of diagnostic reasoning, producing slide-level representations that support both direct diagnostic question-answering and transferable embeddings for downstream tasks. Without additional training, PRISM2 matches or exceeds the cancer-detection performance of clinical-grade products. This is observed without loss of generality on other tasks, where PRISM2 achieves top performance. Finally, using survival prediction as the example, we show that task-specific finetuning with a large dataset can outperform task-specific models, further improving performance. These results demonstrate how language-supervised pretraining provides a scalable, clinically grounded signal for learning generalizable pathology representations, bridging human diagnostic reasoning and foundation-model performance.