Discovering Pathology Rationale and Token Allocation for Efficient Multimodal Pathology Reasoning

TL;DR

This paper introduces a bilateral reinforcement learning framework for multimodal pathology understanding, significantly improving reasoning accuracy and reducing computational costs in diagnostic tasks.

Contribution

It proposes a novel dual-branch approach that enhances reasoning and optimizes token allocation without explicit supervision, advancing multimodal pathology analysis.

Findings

Achieved +41.7% performance improvement

Reduced inference costs by 70.3%

Effective across multiple pathological tasks

Abstract

Multimodal pathological image understanding has garnered widespread interest due to its potential to improve diagnostic accuracy and enable personalized treatment through integrated visual and textual data. However, existing methods exhibit limited reasoning capabilities, which hamper their ability to handle complex diagnostic scenarios. Additionally, the enormous size of pathological images leads to severe computational burdens, further restricting their practical deployment. To address these limitations, we introduce a novel bilateral reinforcement learning framework comprising two synergistic branches. One reinforcement branch enhances the reasoning capability by enabling the model to learn task-specific decision processes, i.e., pathology rationales, directly from labels without explicit reasoning supervision. While the other branch dynamically allocates a tailored number of tokens to different images based on both their visual content and task context, thereby optimizing computational efficiency. We apply our method to various pathological tasks such as visual question answering, cancer subtyping, and lesion detection. Extensive experiments show an average +41.7 absolute performance improvement with 70.3% lower inference costs over the base models, achieving both reasoning accuracy and computational efficiency.