A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension

TL;DR

This paper introduces RCCF, a real-time cross-modality correlation filtering approach for referring expression comprehension, achieving high speed and improved accuracy by reformulating the task as a correlation filtering process.

Contribution

The paper proposes a novel one-stage correlation filtering method that enables real-time inference without accuracy loss, differing from traditional multi-stage approaches.

Findings

Runs at 40 FPS, outperforming existing methods in speed.

Almost doubles the state-of-the-art performance on RefClef dataset.

Achieves leading results on multiple benchmarks including RefCOCO and RefCOCO+.

Abstract

Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.