Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Network

TL;DR

This paper introduces EPNG, a real-time, end-to-end network for Panoptic Narrative Grounding that improves accuracy and speed over existing two-stage methods by using innovative attention and semantic alignment techniques.

Contribution

The paper presents a novel one-stage network with Locality-Perceptive Attention and Semantic Alignment Loss for efficient and accurate PNG, enabling real-time performance.

Findings

Achieves up to 9.4% higher accuracy than baseline

10 times faster inference compared to two-stage models

Demonstrates strong zero-shot generalization on other grounding tasks

Abstract

Panoptic Narrative Grounding (PNG) is an emerging cross-modal grounding task, which locates the target regions of an image corresponding to the text description. Existing approaches for PNG are mainly based on a two-stage paradigm, which is computationally expensive. In this paper, we propose a one-stage network for real-time PNG, termed End-to-End Panoptic Narrative Grounding network (EPNG), which directly generates masks for referents. Specifically, we propose two innovative designs, i.e., Locality-Perceptive Attention (LPA) and a bidirectional Semantic Alignment Loss (SAL), to properly handle the many-to-many relationship between textual expressions and visual objects. LPA embeds the local spatial priors into attention modeling, i.e., a pixel may belong to multiple masks at different scales, thereby improving segmentation. To help understand the complex semantic relationships, SAL proposes a bidirectional contrastive objective to regularize the semantic consistency inter modalities. Extensive experiments on the PNG benchmark dataset demonstrate the effectiveness and efficiency of our method. Compared to the single-stage baseline, our method achieves a significant improvement of up to 9.4% accuracy. More importantly, our EPNG is 10 times faster than the two-stage model. Meanwhile, the generalization ability of EPNG is also validated by zero-shot experiments on other grounding tasks.