HARIS: Human-Like Attention for Reference Image Segmentation

TL;DR

HARIS introduces a novel Human-Like Attention mechanism combined with parameter-efficient fine-tuning to improve referring image segmentation accuracy and zero-shot capabilities, outperforming existing methods on multiple benchmarks.

Contribution

The paper presents a new attention mechanism that focuses on relevant objects and maintains pre-trained encoder abilities using PEFT, advancing RIS performance.

Findings

Achieves state-of-the-art results on RIS benchmarks

Demonstrates strong zero-shot segmentation ability

Outperforms previous methods in accuracy and efficiency

Abstract

Referring image segmentation (RIS) aims to locate the particular region corresponding to the language expression. Existing methods incorporate features from different modalities in a \emph{bottom-up} manner. This design may get some unnecessary image-text pairs, which leads to an inaccurate segmentation mask. In this paper, we propose a referring image segmentation method called HARIS, which introduces the Human-Like Attention mechanism and uses the parameter-efficient fine-tuning (PEFT) framework. To be specific, the Human-Like Attention gets a \emph{feedback} signal from multi-modal features, which makes the network center on the specific objects and discard the irrelevant image-text pairs. Besides, we introduce the PEFT framework to preserve the zero-shot ability of pre-trained encoders. Extensive experiments on three widely used RIS benchmarks and the PhraseCut dataset demonstrate that our method achieves state-of-the-art performance and great zero-shot ability.