PathReasoning: A multimodal reasoning agent for query-based ROI navigation on whole-slide images

TL;DR

PathReasoning is a multimodal reasoning agent that navigates gigapixel whole-slide images by iterative question-guided reasoning, significantly improving ROI selection and diagnostic report accuracy without dense annotations.

Contribution

It introduces a novel multi-modal reasoning framework that mimics pathologist navigation, enabling efficient, interpretable, and annotation-free ROI identification in large-scale pathology images.

Findings

Outperforms existing ROI-selection methods by 6.7% and 3.1% AUROC on key tasks.

Achieves 10% higher accuracy in breast cancer report generation compared to GPT-4o.

Effectively constructs interpretable reasoning chains for digital pathology analysis.

Abstract

Deciphering tumor microenvironment from Whole Slide Images (WSIs) is intriguing as it is key to cancer diagnosis, prognosis and treatment response. While these gigapixel images on one hand offer a comprehensive portrait of cancer, on the other hand, the extremely large size, as much as more than 10 billion pixels, make it challenging and time-consuming to navigate to corresponding regions to support diverse clinical inspection. Inspired by pathologists who conducted navigation on WSIs with a combination of sampling, reasoning and self-reflection, we proposed "PathReasoning", a multi-modal reasoning agent that iteratively navigates across WSIs through multiple rounds of reasoning and refinements. Specifically, starting with randomly sampled candidate regions, PathReasoning reviews current selections with self-reflection, reasoning over the correspondence between visual observations and clinical questions, and concludes by proposing new regions to explore. Across rounds, PathReasoning builds a reasoning chain that gradually directs attention to diagnostically relevant areas. PathReasoning turns each whole slide into a sequence of question-guided views, allowing the model to efficiently find informative ROIs within a fixed number of steps, without the need for dense pixel-level annotations. PathReasoning can substantially outperform strong ROI-selection approaches by 6.7% and 3.1% of AUROC on subtyping and longitudinal analysis tasks. The high-quality ROIs further support accurate report generation on breast cancer, significantly outperforming the standard GPT-4o by 10% in accuracy. PathReasoning prioritizes question-specific regions and constructs interpretable reasoning chains, supporting efficient slide review, consistent diagnostic interpretations, comprehensive reporting, and evidence traceability in digital pathology.