Panoptic Narrative Grounding
C. Gonz\'alez, N. Ayobi, I. Hern\'andez, J. Hern\'andez, J., Pont-Tuset, P. Arbel\'aez

TL;DR
This paper introduces Panoptic Narrative Grounding, a new task combining fine-grained visual segmentation with natural language, supported by a new dataset, metrics, and a baseline method to advance research in detailed image understanding.
Contribution
It presents a novel formulation for visual grounding that integrates panoptic segmentation and natural language, along with an automatic annotation transfer algorithm and a baseline achieving 55.4% Average Recall.
Findings
Baseline achieves 55.4% Average Recall.
New dataset and metrics for panoptic narrative grounding.
Automatic annotation transfer method enhances ground truth quality.
Abstract
This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as stepping stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level by using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. To guarantee the quality of our annotations, we take advantage of the semantic structure contained in WordNet to exclusively incorporate noun phrases that are grounded to a meaningfully related panoptic…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Natural Language Processing Techniques · Domain Adaptation and Few-Shot Learning
