Navigating Gigapixel Pathology Images with Large Multimodal Models

TL;DR

This paper introduces GIANT, a framework enabling large multimodal models to navigate gigapixel pathology images iteratively, significantly improving performance on complex clinical questions compared to traditional methods.

Contribution

We developed GIANT, the first system allowing LMMs to navigate whole-slide images interactively, and released MultiPathQA, a new benchmark for pathology reasoning tasks.

Findings

GIANT outperforms patch- and thumbnail-based baselines.

GPT-5 with GIANT achieves 62.5% accuracy on pathologist-authored questions.

Performance approaches or surpasses specialized pathology models.

Abstract

Despite being widely used to support clinical care, general-purpose large multimodal models (LMMs) have generally shown poor or inconclusive performance in medical image interpretation, particularly in pathology, where gigapixel images are used. However, prior studies have used either low-resolution thumbnails or random patches, which likely underestimated model performance. Here, we ask whether LMMs can be adapted to reason coherently and accurately in the evaluation of such images. In this study, we introduce Gigapixel Image Agent for Navigating Tissue (GIANT), the first framework that allows LMMs to iteratively navigate whole-slide images (WSIs) like a pathologist. Accompanying GIANT, we release MultiPathQA, a new benchmark, which comprises 934 WSI-level questions, encompassing five clinically-relevant tasks ranging from cancer diagnosis to open-ended reasoning. MultiPathQA also includes 128 questions, authored by two professional pathologists, requiring direct slide interpretation. Using MultiPathQA, we show that our simple agentic system substantially outperforms conventional patch- and thumbnail-based baselines, approaching or surpassing the performance of specialized models trained on millions of images. For example, on pathologist-authored questions, GPT-5 with GIANT achieves 62.5% accuracy, outperforming specialist pathology models such as TITAN (43.8%) and SlideChat (37.5%). Our findings reveal the strengths and limitations of current foundation models and ground future development of LMMs for expert reasoning in pathology.