EAVL: Explicitly Align Vision and Language for Referring Image Segmentation

TL;DR

EAVL introduces a dynamic, explicit alignment of vision and language features in referring image segmentation, significantly improving fine-grained text-to-pixel correlation over previous fixed-kernel methods.

Contribution

The paper proposes a novel Vision-Language Aligner that uses dynamic convolution kernels based on input features, enhancing multi-modal feature alignment in RIS.

Findings

Surpasses state-of-the-art on RefCOCO, RefCOCO+, and G-Ref datasets.

Uses dynamic convolution kernels for better feature alignment.

Plug-and-play design for easy integration with existing RIS models.

Abstract

Referring image segmentation (RIS) aims to segment an object mentioned in natural language from an image. The main challenge is text-to-pixel fine-grained correlation. In the previous methods, the final results are obtained by convolutions with a fixed kernel, which follows a similar pattern as traditional image segmentation. These methods lack explicit alignment of language and vision features in the segmentation stage, resulting in suboptimal correlation. In this paper, we introduce EAVL, a method explicitly aligning vision and language features. In contrast to fixed convolution kernels, we introduce a Vision-Language Aligner that aligns features in the segmentation stage using dynamic convolution kernels based on the input image and sentence. Specifically, we generate multiple queries representing different emphases of language expression. These queries are transformed into a series of query-based convolution kernels, which are applied in the segmentation stage to produce a series of masks. The final result is obtained by aggregating all masks. Our method harnesses the potential of the multi-modal features in the segmentation stage and aligns language features of different emphases with image features to achieve fine-grained text-to-pixel correlation. We surpass previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins. Additionally, our method is designed to be a generic plug-and-play module for cross-modality alignment in RIS task, making it easy to integrate with other RIS models for substantial performance improvements.