ReinPath: A Multimodal Reinforcement Learning Approach for Pathology

TL;DR

ReinPath introduces a multimodal large language model for pathology that enhances interpretability and reasoning in diagnostic tasks by integrating images and text, supported by a new high-quality dataset and innovative training strategies.

Contribution

The paper presents a novel multimodal pathology LLM with strong reasoning capabilities and a semantic reward strategy, along with a high-quality VQA dataset for complex reasoning tasks.

Findings

Outperforms state-of-the-art methods on the new dataset.

Achieves high accuracy with only 20% of training data.

Comparable zero-shot classification performance to CLIP.

Abstract

Interpretability is significant in computational pathology, leading to the development of multimodal information integration from histopathological image and corresponding text data.However, existing multimodal methods have limited interpretability due to the lack of high-quality dataset that support explicit reasoning and inference and simple reasoning process.To address the above problems, we introduce a novel multimodal pathology large language model with strong reasoning capabilities.To improve the generation of accurate and contextually relevant textual descriptions, we design a semantic reward strategy integrated with group relative policy optimization.We construct a high-quality pathology visual question answering (VQA) dataset, specifically designed to support complex reasoning tasks.Comprehensive experiments conducted on this dataset demonstrate that our method outperforms state-of-the-art methods, even when trained with only 20% of the data.Our method also achieves comparable performance on downstream zero-shot image classification task compared with CLIP.