A Proposal-Free Query-Guided Network for Grounded Multimodal Named Entity Recognition

TL;DR

This paper introduces a proposal-free, query-guided network for grounded multimodal named entity recognition that improves accuracy by unifying reasoning and decoding through text guidance, outperforming existing models.

Contribution

The proposed QGN model eliminates the need for pre-trained object detectors, enabling more precise grounding of entities in images through integrated multimodal reasoning.

Findings

QGN achieves top performance on benchmark datasets.

It outperforms models using pre-trained detectors.

The approach enhances grounding accuracy in open-domain scenarios.

Abstract

Grounded Multimodal Named Entity Recognition (GMNER) identifies named entities, including their spans and types, in natural language text and grounds them to the corresponding regions in associated images. Most existing approaches split this task into two steps: they first detect objects using a pre-trained general-purpose detector and then match named entities to the detected objects. However, these methods face a major limitation. Because pre-trained general-purpose object detectors operate independently of textual entities, they tend to detect common objects and frequently overlook specific fine-grained regions required by named entities. This misalignment between object detectors and entities introduces imprecision and can impair overall system performance. In this paper, we propose a proposal-free Query-Guided Network (QGN) that unifies multimodal reasoning and decoding through text guidance and cross- modal interaction. QGN enables accurate grounding and robust performance in open-domain scenarios. Extensive experiments demonstrate that QGN achieves top performance among compared GMNER models on widely used benchmarks.