Black-box Attacks on Image Activity Prediction and its Natural Language Explanations

TL;DR

This paper investigates the vulnerability of multimodal, natural language explanation methods in image activity recognition models to black-box adversarial attacks, revealing they can be manipulated with minimal information.

Contribution

It is the first study to evaluate the robustness of multimodal XAI explanations against black-box attacks in image activity recognition models.

Findings

Adversarial perturbations can mislead explanation outputs without affecting predictions.

Black-box attacks require only access to model outputs, not internal details.

Explanations can be manipulated to be unfaithful to the model's true reasoning.

Abstract

Explainable AI (XAI) methods aim to describe the decision process of deep neural networks. Early XAI methods produced visual explanations, whereas more recent techniques generate multimodal explanations that include textual information and visual representations. Visual XAI methods have been shown to be vulnerable to white-box and gray-box adversarial attacks, with an attacker having full or partial knowledge of and access to the target system. As the vulnerabilities of multimodal XAI models have not been examined, in this paper we assess for the first time the robustness to black-box attacks of the natural language explanations generated by a self-rationalizing image-based activity recognition model. We generate unrestricted, spatially variant perturbations that disrupt the association between the predictions and the corresponding explanations to mislead the model into generating unfaithful explanations. We show that we can create adversarial images that manipulate the explanations of an activity recognition model by having access only to its final output.