Generalised Medical Phrase Grounding

TL;DR

This paper introduces MedGrounder, a novel model for generalized medical phrase grounding that maps sentences to multiple image regions, handling complex cases like multi-region findings and negations, improving interpretability of radiological reports.

Contribution

The paper proposes the first GMPG model, MedGrounder, capable of handling zero to multiple regions per phrase, with a new training regime and strong zero-shot transfer capabilities.

Findings

MedGrounder outperforms existing baselines on multi-region and non-groundable phrases.

It achieves strong zero-shot transfer performance.

Uses fewer human annotations than previous methods.

Abstract

Medical phrase grounding (MPG) maps textual descriptions of radiological findings to corresponding image regions. These grounded reports are easier to interpret, especially for non-experts. Existing MPG systems mostly follow the referring expression comprehension (REC) paradigm and return exactly one bounding box per phrase. Real reports often violate this assumption. They contain multi-region findings, non-diagnostic text, and non-groundable phrases, such as negations or descriptions of normal anatomy. Motivated by this, we reformulate the task as generalised medical phrase grounding (GMPG), where each sentence is mapped to zero, one, or multiple scored regions. To realise this formulation, we introduce the first GMPG model: MedGrounder. We adopted a two-stage training regime: pre-training on report sentence--anatomy box alignment datasets and fine-tuning on report sentence--human annotated box datasets. Experiments on PadChest-GR and MS-CXR show that MedGrounder achieves strong zero-shot transfer and outperforms REC-style and grounded report generation baselines on multi-region and non-groundable phrases, while using far fewer human box annotations. Finally, we show that MedGrounder can be composed with existing report generators to produce grounded reports without retraining the generator.