Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

TL;DR

PathSearch is a scalable multimodal retrieval framework that combines fine-grained visual and semantic slide representations using vision-language contrastive learning, significantly improving accuracy and consistency in digital pathology retrieval tasks.

Contribution

This work introduces PathSearch, a novel multimodal retrieval system that unifies detailed visual and semantic slide features for accurate, scalable, and versatile pathology image retrieval.

Findings

Outperforms traditional retrieval methods on multiple datasets

Enhances diagnostic accuracy and inter-observer agreement

Supports both image-to-image and text-to-slide retrieval

Abstract

The rapid digitization of histopathology slides has opened up new possibilities for computational tools in clinical and research workflows. Among these, content-based slide retrieval stands out, enabling pathologists to identify morphologically and semantically similar cases, thereby supporting precise diagnoses, enhancing consistency across observers, and assisting example-based education. However, effective retrieval of whole slide images (WSIs) remains challenging due to their gigapixel scale and the difficulty of capturing subtle semantic differences amid abundant irrelevant content. To overcome these challenges, we present PathSearch, a retrieval framework that unifies fine-grained attentive mosaic representations with global-wise slide embeddings aligned through vision-language contrastive learning. Trained on a corpus of 6,926 slide-report pairs, PathSearch captures both fine-grained morphological cues and high-level semantic patterns to enable accurate and flexible retrieval. The framework supports two key functionalities: (1) mosaic-based image-to-image retrieval, ensuring accurate and efficient slide research; and (2) multi-modal retrieval, where text queries can directly retrieve relevant slides. PathSearch was rigorously evaluated on four public pathology datasets and three in-house cohorts, covering tasks including anatomical site retrieval, tumor subtyping, tumor vs. non-tumor discrimination, and grading across diverse organs such as breast, lung, kidney, liver, and stomach. External results show that PathSearch outperforms traditional image-to-image retrieval frameworks. A multi-center reader study further demonstrates that PathSearch improves diagnostic accuracy, boosts confidence, and enhances inter-observer agreement among pathologists in real clinical scenarios. These results establish PathSearch as a scalable and generalizable retrieval solution for digital pathology.