Panoptic Narrative Grounding

TL;DR

This paper introduces Panoptic Narrative Grounding, a new task combining fine-grained visual segmentation with natural language, supported by a new dataset, metrics, and a baseline method to advance research in detailed image understanding.

Contribution

It presents a novel formulation for visual grounding that integrates panoptic segmentation and natural language, along with an automatic annotation transfer algorithm and a baseline achieving 55.4% Average Recall.

Findings

Baseline achieves 55.4% Average Recall.

New dataset and metrics for panoptic narrative grounding.

Automatic annotation transfer method enhances ground truth quality.

Abstract

This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as stepping stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level by using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. To guarantee the quality of our annotations, we take advantage of the semantic structure contained in WordNet to exclusively incorporate noun phrases that are grounded to a meaningfully related panoptic segmentation region. The proposed baseline achieves a performance of 55.4 absolute Average Recall points. This result is a suitable foundation to push the envelope further in the development of methods for Panoptic Narrative Grounding.