PathVG: A New Benchmark and Dataset for Pathology Visual Grounding

TL;DR

PathVG introduces a new benchmark and dataset for pathology visual grounding, enabling region detection based on complex expressions, with a novel knowledge-enhanced model leveraging LLMs for improved performance.

Contribution

We present PathVG, a novel benchmark and dataset for pathology visual grounding, and propose PKNet, a knowledge-enhanced network utilizing LLMs to handle implicit pathological information.

Findings

PKNet achieves state-of-the-art performance on PathVG.

Implicit information in pathological expressions is a major challenge.

Knowledge enhancement improves visual grounding accuracy.

Abstract

With the rapid development of computational pathology, many AI-assisted diagnostic tasks have emerged. Cellular nuclei segmentation can segment various types of cells for downstream analysis, but it relies on predefined categories and lacks flexibility. Moreover, pathology visual question answering can perform image-level understanding but lacks region-level detection capability. To address this, we propose a new benchmark called Pathology Visual Grounding (PathVG), which aims to detect regions based on expressions with different attributes. To evaluate PathVG, we create a new dataset named RefPath which contains 27,610 images with 33,500 language-grounded boxes. Compared to visual grounding in other domains, PathVG presents pathological images at multi-scale and contains expressions with pathological knowledge. In the experimental study, we found that the biggest challenge was the implicit information underlying the pathological expressions. Based on this, we proposed Pathology Knowledge-enhanced Network (PKNet) as the baseline model for PathVG. PKNet leverages the knowledge-enhancement capabilities of Large Language Models (LLMs) to convert pathological terms with implicit information into explicit visual features, and fuses knowledge features with expression features through the designed Knowledge Fusion Module (KFM). The proposed method achieves state-of-the-art performance on the PathVG benchmark.