Real-Time Referring Expression Comprehension by Single-Stage Grounding Network

TL;DR

This paper introduces a single-stage, end-to-end model called SSG for real-time referring expression comprehension, achieving high accuracy and efficiency without relying on region proposals.

Contribution

The novel SSG model simplifies the process by removing region proposals and introduces a guided attention mechanism and attribute prediction for improved localization.

Findings

Achieves state-of-the-art performance on ReferItGame dataset.

Runs in 25ms per image, significantly faster than previous models.

Performs comparably or better than multi-stage models on benchmark datasets.

Abstract

In this paper, we propose a novel end-to-end model, namely Single-Stage Grounding network (SSG), to localize the referent given a referring expression within an image. Different from previous multi-stage models which rely on object proposals or detected regions, our proposed model aims to comprehend a referring expression through one single stage without resorting to region proposals as well as the subsequent region-wise feature extraction. Specifically, a multimodal interactor is proposed to summarize the local region features regarding the referring expression attentively. Subsequently, a grounder is proposed to localize the referring expression within the given image directly. For further improving the localization accuracy, a guided attention mechanism is proposed to enforce the grounder to focus on the central region of the referent. Moreover, by exploiting and predicting visual attribute information, the grounder can further distinguish the referent objects within an image and thereby improve the model performance. Experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate that our proposed SSG without relying on any region proposals can achieve comparable performance with other advanced models. Furthermore, our SSG outperforms the previous models and achieves the state-of-art performance on the ReferItGame dataset. More importantly, our SSG is time efficient and can ground a referring expression in a 416*416 image from the RefCOCO dataset in 25ms (40 referents per second) on average with a Nvidia Tesla P40, accomplishing more than 9* speedups over the existing multi-stage models.