PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis

TL;DR

PathMR is a multimodal visual reasoning framework that enhances interpretability and diagnostic accuracy in pathology by generating cell-level explanations and predicting lesion patterns, outperforming existing methods.

Contribution

It introduces PathMR, a novel cell-level multimodal reasoning model that improves interpretability and diagnostic performance in pathological image analysis.

Findings

Outperforms state-of-the-art in text generation quality

Achieves higher segmentation accuracy

Demonstrates better cross-modal alignment

Abstract

Deep learning based automated pathological diagnosis has markedly improved diagnostic efficiency and reduced variability between observers, yet its clinical adoption remains limited by opaque model decisions and a lack of traceable rationale. To address this, recent multimodal visual reasoning architectures provide a unified framework that generates segmentation masks at the pixel level alongside semantically aligned textual explanations. By localizing lesion regions and producing expert style diagnostic narratives, these models deliver the transparent and interpretable insights necessary for dependable AI assisted pathology. Building on these advancements, we propose PathMR, a cell-level Multimodal visual Reasoning framework for Pathological image analysis. Given a pathological image and a textual query, PathMR generates expert-level diagnostic explanations while simultaneously predicting cell distribution patterns. To benchmark its performance, we evaluated our approach on the publicly available PathGen dataset as well as on our newly developed GADVR dataset. Extensive experiments on these two datasets demonstrate that PathMR consistently outperforms state-of-the-art visual reasoning methods in text generation quality, segmentation accuracy, and cross-modal alignment. These results highlight the potential of PathMR for improving interpretability in AI-driven pathological diagnosis. The code will be publicly available in https://github.com/zhangye-zoe/PathMR.

Tables

Table 1. Table 1: The reasoning segmentation performance comparison on the GADVR benchmark. “E2E” stands for end-to-end training. † Ov-Seg is a two-stage approach; it first predicts categories, which are then used as prompts for the segmentation. TFLPOs are tested based on 600 × 600 600\times 600 input.

Method	TFLOPs	Validation		Test Overall		Neoplastic		Inflammatory		Connective		Epithelial		Publication
		gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
Ov-Seg†[35]	–	0.356	0.505	0.383	0.515	0.468	0.522	0.365	0.344	0.101	0.132	0.069	0.121	CVPR’23
LISA-7B [23]	11.61	0.402	0.520	0.429	0.549	0.543	0.574	0.392	0.420	0.183	0.221	0.080	0.113	CVPR’24
PixelLM [24]	15.29	0.411	0.530	0.438	0.558	0.550	0.584	0.400	0.429	0.190	0.237	0.098	0.145	CVPR’24
GSVA [27]	15.61	0.367	0.516	0.409	0.520	0.509	0.543	0.355	0.363	0.104	0.187	0.054	0.096	CVPR’24
MMR-7B [31]	11.14	0.587	0.698	0.602	0.714	0.678	0.710	0.536	0.548	0.340	0.384	0.200	0.264	ICLR’25
PathMR-7B	11.14	0.603	0.715	0.629	0.734	0.689	0.722	0.529	0.533	0.351	0.402	0.211	0.287	Ours
LISA-13B [23]	16.84	0.409	0.529	0.436	0.559	0.554	0.583	0.394	0.422	0.183	0.221	0.075	0.101	CVPR’24
MMR-13B [31]	17.21	0.598	0.708	0.613	0.724	0.709	0.739	0.571	0.581	0.400	0.447	0.232	0.328	ICLR’25
PathMR-13B	17.21	0.624	0.723	0.633	0.740	0.694	0.737	0.685	0.726	0.401	0.467	0.389	0.440	Ours

Table 2. Table 2: The reasoning segmentation performance comparison on PathGen benchmark.

Method	Neoplastic		Inflammatory		Connective		Epithelial		Overall
	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
LISA-LLaVA-13B [23]	0.517	0.584	0.477	0.550	0.429	0.507	0.233	0.318	0.484	0.519
MMR-LLaVA-13B‡ [31]	0.573	0.645	0.521	0.607	0.483	0.577	0.304	0.397	0.545	0.626
PathMR-LLaVA-13B (ours)	0.596	0.674	0.537	0.621	0.498	0.597	0.334	0.421	0.560	0.635
PathMR-Qwen (ours)	0.612	0.681	0.551	0.637	0.511	0.607	0.344	0.434	0.579	0.641

Table 3. Table 3: The referring segmentation performance comparison on GADVR dataset.

Methods	Neoplastic		Inflammatory		Connective		Epithelial		Overall
	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
OV-Seg† [35]	0.508	0.564	0.048	0.036	0.076	0.062	0.062	0.058	0.253	0.405
LISA-7B [23]	0.529	0.561	0.383	0.402	0.180	0.210	0.008	0.113	0.337	0.503
PixelLM [24]	0.538	0.571	0.390	0.408	0.186	0.224	0.095	0.139	0.346	0.511
GSVA [27]	0.494	0.551	0.127	0.209	0.093	0.072	0.025	0.089	0.288	0.454
MMR-7B‡ [24]	0.655	0.736	0.474	0.503	0.474	0.498	0.240	0.300	0.527	0.634
PathMR-7B	0.672	0.756	0.491	0.513	0.497	0.522	0.264	0.314	0.539	0.657
LISA-13B [23]	0.540	0.571	0.386	0.412	0.187	0.219	0.076	0.113	0.341	0.517
MMR-13B‡ [24]	0.695	0.724	0.551	0.561	0.394	0.420	0.228	0.290	0.515	0.664
BioMedParse [36]	0.445	0.478	0.260	0.283	0.214	0.232	0.161	0.177	0.254	0.461
PathMR-13B	0.711	0.746	0.571	0.589	0.402	0.431	0.238	0.311	0.530	0.682

Table 4. Table 4: The conversation performance comparison on GADVR dataset.

Methods	Validation		Test
	BLEU-4	F1	BLEU-4	F1
LISA-7B [23]	0.276	0.575	0.278	0.582
LISA-13B [23]	0.279	0.586	0.286	0.591
PixelLM [24]	0.271	0.574	0.268	0.578
GSVA [27]	0.258	0.566	0.261	0.572
MMR-7B [31]	0.265	0.567	0.270	0.584
MMR-13B [31]	0.280	0.576	0.281	0.590
LLaVA-Med [40]	0.284	0.586	0.283	0.591
PathMR (ours)	0.301	0.609	0.302	0.602

Table 5. Table 5: Ablation study of module designs on referring segmentation on GADVR dataset.

Cls. Sup.	Cons.	Neoplastic		Inflammatory		Connective		Epithelial		Overall
		gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
		0.655	0.736	0.474	0.503	0.474	0.498	0.240	0.300	0.527	0.634
✓		0.659	0.740	0.476	0.500	0.474	0.502	0.253	0.310	0.531	0.644
✓	✓	0.672	0.756	0.491	0.513	0.497	0.522	0.264	0.314	0.539	0.657

Equations

$$v_{seg}=v_g+\text{Conv}(v_l),$$

$$\hat{M}=\text{Dec}(\text{Concat}(v_{seg},f_{seg})),$$

$${\mathcal{L}}_{\text{mask}}={\lambda }_1\text{BCE}(\hat{M},M,W)+{\lambda }_2\text{DICE}(\hat{M},M,W),$$

$$\begin{split}L_{\text{con}}=\frac{1}{N}\sum_{i,j}\Bigl{(}&\,\left|p_{i,j}-p_{i+1,j}\right|+\left|p_{i,j}-p_{i-1,j}\right|\\ &+\left|p_{i,j}-p_{i,j+1}\right|+\left|p_{i,j}-p_{i,j-1}\right|\Bigr{)}\end{split}$$

$${\hat{y}}_{txt}=F(Concat(f_{img},f_{txt})).$$

$${\mathcal{L}}_{txt}=CE({\hat{y}}_{txt},y_{txt}).$$

$$\mathcal{L}={\lambda }_{\text{mask}}{\mathcal{L}}_{mask}+{\lambda }_{\text{txt}}{\mathcal{L}}_{txt}+{\lambda }_{\text{con}}{\mathcal{L}}_{con},$$

Full text

{IEEEkeywords}

Cancer diagnosis, multi-modal visual reasoning, pathological interpretability analysis

\IEEEPARstart

Advancements in tissue sectioning and microscopic imaging have enabled the digitization of whole slide images (WSIs), laying the foundation for computational pathology. Currently, H&E-stained pathological image analysis is widely regarded as the gold standard for cancer diagnosis [1], and prognostic assessment [2]. Meanwhile, deep learning-based cancer diagnosis models have improved pathological evaluations’ efficiency, consistency, and scalability, offering significant advantages over traditional cancer diagnosis approaches.

Despite these advancements, most automated diagnostic methods have primarily focused on improving accuracy while overlooking the diagnostic interpretability for clinical decisions. Although some multiple instance learning (MIL) approaches introduce an attention mechanism to highlight diagnostically relevant regions [3, 4] by computing a global attention map as shown in Fig. 1 (a). However, these methods fail to capture tumor immune characteristics due to patch-level granularity. First, each patch typically contains hundreds of cells with diverse morphologies and spatial arrangements, making it challenging to capture fine-grained details. Second, attention scores reflect relative importance but do not provide explicit textual explanations, limiting their utilization in clinical decision-making. In contrast, pathologists usually assess tumor progression and differentiation by identifying tumor regions and microscopic features, such as cell morphology and their spatial distribution. These cellular-level characteristics provide essential insights into the tumor immune microenvironment, which is critical for effective diagnosis and treatment planning. Therefore, developing a fine-grained interpretable model that integrates visual and textual information is crucial for pathology diagnosis.

Recently, in natural scene understanding, visual reasoning models such as LISA and Groma have integrated large language models with specialized visual perception modules to produce segmentation masks at the pixel level and semantically aligned text in one unified process. Each $<$seg$>$ token is explicitly tied to a corresponding image region, ensuring that generated descriptions map precisely to visual features. This synchronous generation framework extends seamlessly to medical diagnosis. As shown in Fig. 1 (b), the visual branch accurately delineates lesion areas while the language branch generates expert level diagnostic reports. Through careful modular design, masks and narratives remain tightly coupled both spatially and semantically, providing decision support for AI assisted pathology that is transparent and traceable.

Despite these advances, current visual reasoning frameworks often produce fragmented segmentation outputs, evident as shape distortions and boundary noise. These artifacts stem from misalignment between visual and textual features, which allows spurious predictions to persist. To overcome this, we introduce a novel constraint mechanism in the visual perception module that enforces morphological consistency and suppresses isolated noisy predictions. By applying this mechanism, our model yields cleaner, more coherent masks that align more closely with generated text, enhancing interpretability and reliability in pathological image analysis.

This paper presents PathMR, a cell level multimodal visual reasoning model resilient to fragmentation, designed for interpretable pathology diagnosis. Given a histopathology image and an associated text query, PathMR produces expert level narrative responses alongside cell segmentation and classification masks, ensuring precise semantic alignment between visual and textual outputs. To enable standardized evaluation and foster further research, we introduce GADVR, the first pixel level visual reasoning dataset in pathology, which comprises approximately 190,000 image patches annotated by gastric adenocarcinoma subtype and over 550,000 paired image and text entries. Finally, to address fragmentary segmentation in pixel level reasoning tasks, we incorporate class supervision and consistency constraints that reduce uncertainty in cell category predictions and preserve the integrity of cell morphology.

In general, our contributions are summarized as follows:

• (1)

We present a novel pixel-level visual reasoning model for pathological image analysis, offering fine-grained cell-level interpretability to improve clinical diagnosis.

• (2)

To facilitate evaluation, we develop a comprehensive pixel-level multimodal visual reasoning dataset, which includes pixel-wise nuclei segmentation and classification labels alongside cell-level question-answer pairs.

• (3)

We propose a dual constraint mechanism that combines classification supervision and a morphological consistency constraint to stabilize cell class predictions and reduce boundary noise, improving the accuracy of pixel level segmentation in visual reasoning models.

• (4)

Extensive experiments across various visual reasoning models demonstrate that our approach consistently outperforms existing state-of-the-art methods in segmentation accuracy and text generation quality.

1 Related Work

1.1 Interpretable Pathological Diagnosis Models

Deep learning has substantially advanced automated pathological image analysis [5, 6, 7]. However, its clinical integration remains limited, largely due to the lack of interpretability and transparency in model decision-making. To address this, attention-based methods have been developed to provide patch-level interpretability by assigning importance scores to image regions. In particular, multiple instance learning (MIL) frameworks [8, 4, 3] utilize attention mechanisms to highlight diagnostically relevant lesion areas. While effective in localizing salient regions, these approaches often produce coarse attention maps that lack clinical validation and do not reflect the reasoning process of pathologists. To move beyond attention, multimodal strategies have been proposed to enhance interpretability by aligning image features with textual information. For instance, Zhang et al. [9] linked histopathology patches to textual descriptions, enabling a more human-understandable model output. Others have focused on quantifying cellular-level feature distributions across tumor stages [10, 11], aiming to provide statistical insights into disease progression. Despite these advances, most existing methods still lack fine-grained interpretability, highlighting the importance of developing interpretability frameworks that operate at the microscopic level to reflect clinical reasoning.

1.2 Multimodal Pathology Models

Early vision language efforts in computational pathology demonstrated that gigapixel scale WSIs can be used for downstream question answering and report generation. Chen et al. introduced WSI-VQA, the first framework to perform slide level visual question answering [12], and WSICaption, which generates diagnostic captions directly from WSIs[13], proving that transformer based models can attend across billions of pixels to produce clinically meaningful outputs. Building on this foundation, PathGen[14] shifted the focus to patch level semantics by generating fine grained descriptions for millions of tissue regions to capture nuanced morphological patterns, and HistGen [15] further scaled patch level captioning across diverse datasets to improve descriptive accuracy and granularity. Meanwhile, unified generative frameworks such as CONCH [16], Virchow [17] and CHIEF [6] expanded the scope of multimodal pathology modeling by jointly learning to classify diagnoses, localize regions of interest and generate both slide level and patch level narratives within a single architecture. Most recently, GigaPath[18] and mSTAR[19] have combined extensive WSI collections with multimodal reasoning objectives to achieve robust generalizable performance across a wide range of tissue types and clinical tasks.

1.3 Pixel-Level Visual Reasoning Models

Most large language models (LLMs) are primarily designed to generate global textual descriptions and often lack the capacity for fine-grained, pixel-level visual reasoning [20, 21]. To address this limitation, recent studies have introduced pixel-level supervision and enhanced vision-language alignment mechanisms to improve spatial reasoning capabilities [22]. For example, LISA [23] was one of the first models to enable pixel-level visual reasoning, although it is constrained to single-object understanding. Subsequent models have extended these capabilities to handle more complex scenarios involving multiple instances. PixelLM [24], GlaMM [25], PerceptionGPT [26], and GSVA [27] support multi-object reasoning through improved multimodal alignment and spatial localization. Beyond 2D reasoning, emerging models such as ZSVG3D [28] and PRIMA [29] explore 3D spatial understanding within multimodal frameworks. In parallel, models like OMG-LLaVA [22] and GeoPixel [30] continue to advance pixel-level grounded visual reasoning, pushing the boundaries of interpretability in vision-language models.

Despite their potential for medical imaging, no existing work applies pixel-level visual reasoning to pathology. To bridge this gap, we propose the first multimodal visual reasoning model for pathological images, enhancing both interpretability and diagnostic reliability.

2 Dataset Construction

The advancement of visual reasoning tasks has led to the development of numerous pixel-level reasoning datasets in natural scenes, such as MUSE [24], MMR [31], and ReasonSeg [23]. However, extending pixel-level reaoning models to cancer diagnosis remains highly challenging, primarily due to the difficulty of obtaining suitable datasets. An ideal dataset must meet the following key criteria: (1) Comprehensive cancer diagnostic annotations, including class labels for cancer staging or subtype classification; (2) A large-scale collection of pathological image patches to support the training of visual reasoning models; (3) Pixel-level annotations for muclei segmentation and classification, enabling fine-grained, cell-level pathological analysis; (4) Expert-annotated textual question-answer pairs that align with nuclear annotations and provide diagnostic reasoning in a structured format.

To address these challenges and advance visual reasoning in pathology, we introduce the Gastric Adenocarcinoma Diagnosis Visual Reasoning (GADVR) dataset. Unlike existing cancer diagnosis datasets that primarily provide WSI-level or patient-level labels, GADVR incorporates both pixel-level nuclei annotations and patch-level text annotations, enabling the development of cell-level multimodal reasoning models for fine-grained interpretability analysis.

2.1 Cancer Diagnosis Data

Existing multimodal cancer diagnosis datasets primarily provide patch-level annotations but often lack nuclear segmentation and classification labels. Conversely, datasets designed for nuclear segmentation and classification typically contain only a limited number of image patches. For example, PanNuke [32] consists of only approximately 7,000 patches, each with a resolution of $256\times 256$ pixels. These limitations hinder the development of large-scale visual reasoning models.

To address these challenges, we construct the GADVR dataset based on PatchGastricADC22 [33], a dataset for gastric adenocarcinoma subtype diagnosis. PatchGastricADC22 consists of 262,777 pathological image patches, each with a resolution of $600\times 600$ pixels at $\times$40 magnification. These patches are categorized into nine gastric adenocarcinoma subtypes based on their corresponding WSI labels, including well-differentiated, moderately differentiated tubular adenocarcinoma, and papillary adenocarcinoma, among others. The large dataset size and comprehensive subtype annotations provide a strong foundation for developing multimodal reasoning models. Additionally, PatchGastricADC22 includes WSI-level diagnostic reports, which serve as a valuable resource for generating text-based question-answer pairs.

2.2 GADVR Generation Pipeline

Our dataset construction process includes: 1) pixel-level nuclear segmentation, classification label generation, and 2) cell-level text question-answer (QA) pair generation. The specific process is shown in Fig. 2.

2.2.1 Nuclei Annotation Generation

Since patch-level labels are inherited from WSI annotations, certain patches may exhibit weak correlations between the local tissue texture and the assigned subtype label. To address this, we first apply a pretrained CLIP-ViT model [14], leveraging the corresponding original diagnostic reports to identify and remove weakly correlated patches. The remaining high-correlation patches are then used to generate pixel-level nuclei segmentation and classification labels using a pretrained HoverNet model [34]. To determine the most suitable pretraining dataset for HoverNet, we evaluate models trained on several publicly available datasets, including ConSeP, Lizard, and PanNuke. Based on expert pathological review, PanNuke is selected for pretraining due to its superior segmentation performance.

To further improve nuclei classification accuracy, two pathologists manually correct misclassified nuclei in approximately 200 patches per category. These corrections are integrated into a human-in-the-loop refinement process to optimize the classification labels. Subsequently, three independent pathologists assess the final nuclei segmentation and classification results, confirming that the constructed dataset achieved excellent quality. A detailed quality assessment is provided in the subsection Quality Control.

2.2.2 Question-Answer Pair Generation

To enable fine-grained interpretability, our dataset incorporates cell-level question-answer (QA) pairs designed to support pixel-wise pathological reasoning. The data generation process is illustrated in Fig. 2.

Question Generation We employ GPT-4o to generate a diverse set of diagnostic questions based on pathology reports and specific cancer subtypes. The questions are categorized into two types: (1) General questions, which focus on describing pathological image features such as nuclear morphology, spatial distribution, and tissue architecture; and (2) Subtype-specific questions, which target diagnostic reasoning, such as inferring the cancer subtype from nuclear morphology and distribution patterns. These two categories are complementary and progressive, where understanding general morphological characteristics provides a foundation for accurate subtype diagnosis. In total, approximately 200 questions are generated.

Answer Generation For each image, three questions are randomly sampled from the question pool and answered using GPT-4o. To improve clinical relevance and contextual accuracy, both the selected questions and the corresponding nuclei segmentation masks are provided as inputs to the model. The prompt design is iteratively refined to guide the model in generating answers that explicitly incorporate nuclear categories and spatial distribution patterns. This ensures that the generated responses are closely aligned with the visual information conveyed by the segmentation masks.

2.3 Dataset Presentation

Fig. 3 illustrates representative examples from the GADVR dataset, which integrates cell-level image-text annotations to support interpretable diagnosis of gastric adenocarcinoma. The dataset is designed to facilitate three core tasks: reasoning segmentation, referring segmentation, and conversation task. In the reasoning segmentation, the model aligns pixel-level visual predictions with textual answers, enabling explicit visual reasoning. The model not only localizes cells, but also associates them with corresponding textual descriptions that explain their diagnostic significance. The referring segmentation focuses on identifying specific nuclei types based on given textual references. Finally, the conversation task enhances interpretability by simulating a dialogue grounded in pathology knowledge.

2.4 Quality Control

To assess the quality of the constructed GADVR dataset, we invited three independent external pathologists to evaluate a randomly selected subset of 200 image samples. Each sample includes nuclei segmentation masks and their corresponding QA pairs. For each sample, the pathologists assessed the overall image quality (good, okay, or bad) and verified whether the key structures were correctly segmented and free of major artifacts or errors. In addition, the quality of QA pairs was evaluated with respect to their clinical meaningflness, image dependency and correctness. Fig. 4 summarizes the evaluation results. The overall distribution remains consistent across the three experts. Specifically, over 92% of the segmentation masks are rated as Correct, and more than 95% of the QA pairs are judged to be of Okay or Good quality. These results demonstrate that the dataset maintains high quality in both segmentation accuracy and textual reasoning, making it a reliable resource for interpretable pathology model development.

3 Methods

Although multimodal large language models (LLMs) can enhance feature extraction through vision-language interaction, existing methods typically support only unimodal text output and lack the capability for pixel-level reasoning. To address this limitation, we introduce an visual perception module on top of the existing LLM framework, enabling mask prediction for images. In detail, the architecture of our proposed model is illustrated in Fig. 5.

3.1 Visual Perception Module

Vision Backone The visual perception module employs a vision backbone to extract deep image features, as illustrated in the blue box in Fig. 5. We utilize the SAM vision encoder to capture multi-scale representations from the input image $x_{img}\in\mathbb{R}^{h\times w\times 3}$. To effectively aggregate both global and local information, we incorporate the feature aggregation layers proposed in MMR [31]. Specifically, the global feature representation is denoted as $v_{g}\in\mathbb{R}^{h/16\times w/16\times c}$, while the early-stage local features are represented as $v_{l}\in\mathbb{R}^{h/16\times w/16\times c^{\prime}}$, where $h$ and $w$ are the height and width of the input image, and $c$ and $c^{\prime}$ denote the channel dimensions. The final segmentation feature representation $v_{seg}$, is computed as follows:

[TABLE]

where $\text{Conv}(\cdot)$ denotes convolutional operations applied to the local feature representation.

Vision Decoder The vision decoder takes as input both the segmentation feature $v_{seg}$ from the vision backbone and the feature representation $f_{seg}$ obtained from the MLLM module to generate the final segmentation mask:

[TABLE]

where $\text{Dec}(\cdot)$ represents the decoder function, and $\text{Concat}(\cdot)$ denotes the concatenation operation.

Classification Supervision In visual reasoning tasks, pixel-level segmentation results often contain significant prediction noise, including isolated prediction masks and irregular nuclei morphologies. To address the isolated segmentation results caused by uncertainty in category predictions, we introduce a classification supervision method before calculating the segmentation loss, as shown in Fig. 5 (b).

In the design, the text output $\hat{y}_{txt}$ undergoes token filtering to extract the representations corresponding to the $<seg>$ token, yielding $\hat{y}_{fil}$. This filtered representation is then mapped to the MLLM output $f_{\text{seg}}$, which serves as input for the visual decoder. However, during this process, the network incrementally predicts the $<seg>$ token and outputs corresponding binary segmentation masks $\hat{M}_{i}$. This sequential prediction leads to a lack of awareness of subsequent categories, causing the model to classify other categories as the current one incorrectly. To mitigate this issue, we apply classification supervision by introducing additional penalties for incorrect predictions when the model mistakenly identifies other categories as the current class. This ensures greater accuracy in category predictions. The segmentation loss in this process is defined as:

[TABLE]

where $\text{BCE}(\cdot)$ and $\text{DICE}(\cdot)$ represent the binary cross-entropy and Dice loss functions, respectively. $W$ represents the penalty weight set to 1.5, and the hyperparameters $\lambda_{1}$ and $\lambda_{2}$ are set to 2 and 0.5 in the paper.

Consistency Constraint To enhance the completeness of nuclear morphology, we introduce a consistency constraint that enforces adjacent pixels to have similar category predictions. In this step, for each pixel, we evaluate whether its predicted category is consistent with that of its four immediate neighbors (up, down, left, and right). This approach aims to smooth the prediction map, thereby reducing local noise and discontinuities. Specifically, let $p_{ij}$ denote the prediction probability of the pixel at position $(i,j)$. The consistency loss is defined as:

[TABLE]

where $N$ represents the total number of valid neighboring pixel pairs. The objective of this loss is to minimize the differences in prediction probabilities between neighboring pixels, thus ensuring a more consistent class distribution across the predicted map.

3.2 Multimodal Large Language Model

To integrate textual and visual modalities, the input image $x_{img}$ is first processed by a CLIP vision encoder and subsequently mapped into the LLM feature space, producing visual token embeddings $f_{img}\in\mathbb{R}^{N_{img}\times d}$. For simplicity, the mapping layer is omitted in Fig. 5. Meanwhile, the textual input $x_{txt}$ is tokenized and embedded using the language model’s tokenizer, yielding text token embeddings $f_{txt}\in\mathbb{R}^{N_{txt}\times d}$. Then, the visual and textual embeddings are then concatenated and fed into the large language model $F$, which generates the textual response:

[TABLE]

Notably, $\hat{y}_{txt}$ includes both the generated textual response and specialized [SEG] tokens that correspond to segmented objects. Based on this output, the textual loss $L_{txt}$ is computed as follows:

[TABLE]

Finally, the model is optimized end-to-end using a combined objective function that integrates both loss in text generation $\mathcal{L}_{\text{txt}}$ and loss in consistency constraint $\mathcal{L}_{\text{con}}$:

[TABLE]

where the coefficients $\lambda_{\text{mask}}$, $\lambda_{\text{txt}}$ and $\lambda_{\text{con}}$ are used to balance the losses. By default, these weights are set to 1.

4 Experiments

4.1 Datasets

We evaluate the performance of our model on two datasets: the proposed GADVR dataset and the public PathGen dataset. The GADVR111The GADVR dataset is publicly available at: https://huggingface.co/datasets/zhangye-zoe/GADVR contains approximately 190,000 pathology image patches and over 547,000 image-text pairs, all derived from WSIs of gastric adenocarcinoma. To ensure consistent evaluation, we split the dataset by WSI into training, validation, and test subsets following an 8:1:1 ratio. Specifically, the training set contains 149,928 patches and 447,170 image-text pairs, the validation set contains 14,814 patches and 44,080 pairs, and the test set includes 18,754 patches and 56,067 pairs.

To evaluate the generalizability of our model across diverse tissue types, we further conduct training using the PathGen dataset [14], a pan-cancer resource that includes pathology images from over 25 different organs. We randomly sample 50,000 images from PathGen to form an independent training set. Since PathGen does not provide cell-level segmentation annotations, we generate segmentation labels using the same pipeline employed for GADVR. This allows our model to be trained and evaluated consistently across both datasets in terms of cell-level segmentation and image-text reasoning.

4.2 Implementation Details

For the visual perception module, we employ ViT-H SAM [37] as both the visual encoder and decoder. Within the multimodal LLM framework, we use a pre-trained CLIP-ViT-L/14-336 model [38] as the vision encoder, while LLaVA-v0.5-7B and llava-llama-2-13B-chat-lightning-preview [39] serve as the language models. To improve fine-tuning efficiency, we integrate LoRA. Following the training setup of LISA [23], we use the AdamW optimizer with a learning rate of 0.0003. For the 7B model, we set the batch size to 16 with 1200 steps per epoch, while for the 13B model, we use a batch size of 6 with 3125 steps per epoch. The LoRA rank is fixed at 8. Training is conducted on 8 A100 GPUs, with a total training time of approximately 1 day for the 7B model and 1.5 days for the 13B model.

4.3 Baselines and Evaluation Metrics

Baselines To thoroughly assess our model’s performance across reasoning segmentation, referring segmentation, and dialogue tasks, we benchmark it against state-of-the-art visual reasoning models designed for natural image domains, including Ov-Seg [35], LISA [23], PixelLM [24], GSVA [27], and MMR [31], all of which support these three tasks. Additionally, to enable a comprehensive comparison with models tailored for pathology image analysis, we incorporate the multimodal LLMs BiomedParse [36] and LLaVA-Med [40] as additional baselines for the referring segmentation and conversation tasks. To ensure a fair comparison, we fine-tune all visual reasoning models on our dataset.

Evaluation Metrics We evaluate our model’s performance on both segmentation and text generation tasks. For segmentation, we use gIoU and cIoU as evaluation metrics, following prior works [24, 31]. For text conversation, we assess performance using BLEU-4 and F1 scores. Throughout the paper, we denote the best performing result in ** bold** and the second-best result with underline.

4.4 Comparative Experiments

4.4.1 Results on Reasoning Segmentation

Table 1 presents a comprehensive comparative analysis of our method against state-of-the-art visual reasoning approaches. In this experiment, we evaluate models at both the 7B and 13B parameter scales and benchmark their performance against existing methods. The results demonstrate that our approach consistently achieves superior performance across multiple evaluation metrics. Furthermore, to assess the generalization capability of our method across diverse tissue types, we conducte additional experiments on the PathGen dataset, incorporating LLaVA and Qwen as the underlying large language models. As shown in Table 2, our approach maintains a notable advantage over LISA and MMR, achieving improvements of more than 1.5% and 2.4% in the gIoU metric, respectively.

Furthermore, Fig. 6 presents a qualitative comparison of visual reasoning outputs across different models. The visualization demonstrates that our method achieves superior alignment between textual descriptions and segmentation results. Specifically, our model accurately identifies tumor cells, inflammatory cells, and connective tissue, ensuring consistency between the generated response and the segmentation mask. In contrast, other methods exhibit notable limitations: LISA and PixelLM fail to capture crucial regions, while GSVA produces less coherent segmentation. Our approach not only addresses these shortcomings but also provides a more comprehensive and precise interpretation.

4.4.2 Results on Referring Segmentation

To evaluate the effectiveness of our model in referring segmentation, we conduct comprehensive experiments on the GADVR benchmark, and report the results in Table 3. Our proposed PathMR model achieves state-of-the-art performance across all categories on both 7B and 13B scales. Specifically, PathMR-13B attains the highest overall scores of 0.530 gIoU and 0.682 cIoU, surpassing the previous best model MMR-13B, which scores 0.515 and 0.664, respectively. In particular, PathMR demonstrates strong robustness on rare categories, such as epithelial cells, which many baseline methods including LISA and OV-Seg fail to segment accurately.

In addition to the quantitative gains, qualitative comparisons in Fig. 7 further highlight PathMR’s superior reasoning ability. Compared to existing methods, our model yields sharper, more complete, and category-aligned segmentation masks, particularly in regions with dense tumor cells or ambiguous boundaries. Unlike other approaches that tend to under-segment or generate noisy masks, PathMR produces precise predictions that better align with the ground truth.

4.4.3 Results on Conversation

To assess the model’s ability in conversation tasks, we evaluate PathMR on both the validation and test sets of the GADVR benchmark. The task involves answering diagnostic questions based on input pathology images, with optional visual references provided for context. We report the quantitative results in Table 4, and provide a qualitative comparison with LLaVA-Med in Fig. 9.

Our proposed PathMR model achieves the best performance across all metrics, with a BLEU-4 score of 0.301/0.302 and an F1 score of 0.609/0.602 on the validation/test sets, respectively. These results outperform all prior methods, including LLaVA-Med and LISA-13B, both of which rely on large-scale pretraining. The qualitative results in Fig. 9 further highlight the superior interpretability and pathology awareness of our model. In both examples, PathMR provides responses that are factually consistent with the visual features in the image, correctly identifying the cancer subtype and the presence of significant cellular atypia. In contrast, LLaVA-Med fails to detect key diagnostic features or gives vague or incorrect answers, such as misclassifying the cancer subtype or overlooking morphological abnormalities.

4.5 Ablation Studies

To assess the contribution of key components in our framework, we perform ablation experiments on two modules: class supervision and consistency constraint, with results shown in Fig. 8 (reasoning segmentation) and Table 5 (referring segmentation). For the reasoning segmentation, incorporating class supervision improves feature discrimination and enhances category-specific predictions, increasing gIoU from 0.602 to 0.609. The consistency constraint alone also provides a notable improvement by stabilizing predictions in spatially fragmented or morphologically ambiguous regions, reaching gIoU 0.605 and cIoU 0.724. When both modules are applied together, the model achieves the best overall performance with gIoU 0.629 and cIoU 0.734, suggesting that both modules work together to enhance the overall performance.

For the referring segmentation task, although the baseline model performs reasonably well, adding class supervision brings clear improvements, particularly in challenging categories such as epithelial cells. Incorporating the consistency constraint further refines the predictions, especially in morphologically diverse regions like inflammatory tissues. When both modules are combined, the method achieves the highest overall performance, with gIoU reaching 0.539 and cIoU reaching 0.657, delivering consistent gains across all tissue types. These results demonstrate that integrating both modules effectively enhances segmentation accuracy and robustness.

5 Conclusions and Discussion

In this work, we introduced PathMR, a novel multimodal visual reasoning framework designed to enhance interpretability in computational pathology. By integrating pixel-level segmentation with language-based diagnostic reasoning, PathMR enables transparent and clinically meaningful decision support. Comprehensive experiments on two benchmark datasets demonstrate that PathMR consistently surpasses state-of-the-art visual reasoning models. Looking ahead, we plan to incorporate tissue-level segmentation labels to improve the spatial localization of nuclei within their anatomical context, enabling more fine-grained and structure-aware diagnostic reasoning. Additionally, we aim to expand PathMR to a broader range of cancer types and tissue morphologies, validating its scalability and robustness across diverse, large-scale datasets. These enhancements will further advance PathMR toward integration into real-world clinical workflows, offering pathologists interpretable, reliable, and evidence-based diagnostic support.