Proxy-Embedding as an Adversarial Teacher: An Embedding-Guided Bidirectional Attack for Referring Expression Segmentation Models

TL;DR

This paper introduces PEAT, a novel embedding-guided bidirectional adversarial attack method that exposes vulnerabilities in referring expression segmentation models, enhancing their robustness and security against diverse and sensitive inputs.

Contribution

The paper proposes PEAT, a new adversarial attack technique specifically designed for RES models, addressing their multimodal structure and generalization across varied expressions.

Findings

PEAT outperforms existing baselines across multiple RES architectures.

The attack reveals significant vulnerabilities in current RES models.

Experiments demonstrate improved robustness and security of RES systems.

Abstract

Referring Expression Segmentation (RES) enables precise object segmentation in images based on natural language descriptions, offering high flexibility and broad applicability in real-world vision tasks. Despite its impressive performance, the robustness of RES models against adversarial examples remains largely unexplored. While prior adversarial attack methods have explored adversarial robustness on conventional segmentation models, they perform poorly when directly applied to RES models, failing to expose vulnerabilities in its multimodal structure. In practical open-world scenarios, users typically issue multiple, diverse referring expressions to interact with the same image, highlighting the need for adversarial examples that generalize across varied textual inputs. Furthermore, from the perspective of privacy protection, ensuring that RES models do not segment sensitive content without explicit authorization is a crucial aspect of enhancing the robustness and security of multimodal vision-language systems. To address these challenges, we present PEAT, an Embedding-Guided Bidirectional Attack for RES models. Extensive experiments across multiple RES architectures and standard benchmarks show that PEAT consistently outperforms competitive baselines.