Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

TL;DR

Path-RAG introduces a knowledge-guided retrieval framework using HistoCartography to enhance open-ended pathology visual question answering, significantly improving accuracy by leveraging domain-specific tissue and cell information.

Contribution

The paper presents a novel Path-RAG framework that incorporates domain knowledge retrieval via HistoCartography, addressing the limitations of deep learning in complex pathology image analysis.

Findings

Improves PathVQA-Open accuracy from 38% to 47%.

Achieves 28% gain on H&E-stained pathology images.

Enhances long-form question answering performance by over 30%.

Abstract

Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).