Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition

TL;DR

This paper introduces the first adversarial attack and defense framework for pedestrian attribute recognition, exploiting semantic and label perturbations to evaluate and improve model robustness across digital and physical domains.

Contribution

It proposes a novel adversarial attack and defense framework for PAR using multi-modal transformers and CLIP, addressing vulnerability issues in pedestrian attribute recognition models.

Findings

Effective adversarial attacks on PAR models demonstrated

Semantic offset defense improves robustness against attacks

Validated on multiple digital and physical datasets

Abstract

Pedestrian Attribute Recognition (PAR) is an indispensable task in human-centered research and has made great progress in recent years with the development of deep neural networks. However, the potential vulnerability and anti-interference ability have still not been fully explored. To bridge this gap, this paper proposes the first adversarial attack and defense framework for pedestrian attribute recognition. Specifically, we exploit both global- and patch-level attacks on the pedestrian images, based on the pre-trained CLIP-based PAR framework. It first divides the input pedestrian image into non-overlapping patches and embeds them into feature embeddings using a projection layer. Meanwhile, the attribute set is expanded into sentences using prompts and embedded into attribute features using a pre-trained CLIP text encoder. A multi-modal Transformer is adopted to fuse the obtained vision and text tokens, and a feed-forward network is utilized for attribute recognition. Based on the aforementioned PAR framework, we adopt the adversarial semantic and label-perturbation to generate the adversarial noise, termed ASL-PAR. We also design a semantic offset defense strategy to suppress the influence of adversarial attacks. Extensive experiments conducted on both digital domains (i.e., PETA, PA100K, MSP60K, RAPv2) and physical domains fully validated the effectiveness of our proposed adversarial attack and defense strategies for the pedestrian attribute recognition. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR.