Spatial Semantic Recurrent Mining for Referring Image Segmentation

TL;DR

This paper introduces Spatial Semantic Recurrent Mining (S²RM), a novel method for Referring Image Segmentation that enhances cross-modality fusion by recurrently correlating semantic features across spatial and contextual dimensions.

Contribution

The paper proposes S²RM, a new spatial semantic recurrent framework, and a Cross-scale Abstract Semantic Guided Decoder (CASG) for improved referent segmentation accuracy.

Findings

Outperforms state-of-the-art on four challenging datasets.

Effectively models global relationships and structured semantics.

Enhances cross-modality feature fusion in RIS.

Abstract

Referring Image Segmentation (RIS) consistently requires language and appearance semantics to more understand each other. The need becomes acute especially under hard situations. To achieve, existing works tend to resort to various trans-representing mechanisms to directly feed forward language semantic along main RGB branch, which however will result in referent distribution weakly-mined in space and non-referent semantic contaminated along channel. In this paper, we propose Spatial Semantic Recurrent Mining (S\textsuperscript{2}RM) to achieve high-quality cross-modality fusion. It follows a working strategy of trilogy: distributing language feature, spatial semantic recurrent coparsing, and parsed-semantic balancing. During fusion, S\textsuperscript{2}RM will first generate a constraint-weak yet distribution-aware language feature, then bundle features of each row and column from rotated features of one modality context to recurrently correlate relevant semantic contained in feature from other modality context, and finally resort to self-distilled weights to weigh on the contributions of different parsed semantics. Via coparsing, S\textsuperscript{2}RM transports information from the near and remote slice layers of generator context to the current slice layer of parsed context, capable of better modeling global relationship bidirectional and structured. Besides, we also propose a Cross-scale Abstract Semantic Guided Decoder (CASG) to emphasize the foreground of the referent, finally integrating different grained features at a comparatively low cost. Extensive experimental results on four current challenging datasets show that our proposed method performs favorably against other state-of-the-art algorithms.